Method and apparatus for registration and comparison of medical images

ABSTRACT

Methods and apparatuses disclosed herein process medical images, for comparison and analysis of the images. The method according to one embodiment accesses digital image data representing a first medical image and a second medical image; registers the second image to the first image using a specific region preserving registration or specific regions preserving registration, to obtain a registered second image; and compares the first image and the registered second image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a digital image processing technique,and more particularly to a method and apparatus for registering images.

2. Description of the Related Art

Comparative analysis of medical images is typically performed to observeanatomical changes, to identify abnormal growth, to observe impact oftreatment, etc. For example, comparison of mammograms may help identifyabnormal structures and diagnose medical problems in breasts. Temporalcomparison is an important tool in the analysis of mammograms. Temporalcomparison is useful, for example, in cases where it is difficult todetect cancers without prior mammograms of the patient. In such cases,it is easier to detect cancers in current mammograms by comparingcurrent mammograms with prior mammograms of the patient.

In analog screening mammography, temporal comparison is typicallyimplemented by arranging mammograms side by side, or one above theother. A reader (for example, a radiologist) moves his head up and downor left and right to compare each region of interest (ROI) in currentand prior mammograms. In the case of analog film mammograms, this manualcomparison method is currently the best technique to perform a temporalcomparison of mammograms. Major drawbacks, however, are associated withthis method. For example, a large eye movement is needed to comparemammograms arranged side by side or one above the other. In addition, itis difficult to spot differences between mammograms arranged as such,and it is virtually impossible to hang more than 2 or 3 temporal cases(mammograms) for comparison, due to spatial constraints. Moreover,variations in breast positioning differences, mammogram background,etc., complicate the determination of meaningful differences betweenmammogram images.

While analog mammography has been gradually replaced by computedradiography (CR) and full field digital mammography (FFDM), hardcopyreading using light boxes has been replaced by softcopy reading usingPicture Archiving and Communications Systems (PACS) and mammographyworkstations. Despite these advances, however, mammogram reading ofimages placed side by side is still extensively used by medicalprofessionals because it is easily implemented.

Another method for temporal comparison of mammograms uses digital imageprocessing to subtract a current mammogram image from a correspondingprior image. Temporal subtraction has been applied to temporalcomparison of chest X-ray images, where it detects subtle differencesbetween X-ray images. The temporal subtraction technique applied tochest X-ray images does not, however, work for mammograms, becauseunlike chest X-ray images, breast images are hard to align. This is sobecause breasts are deformable and contain no obvious landmarks,especially when they are compressed in a mammography machine. Hence,subtraction of breast images using the method that was applied to chestX-ray images does not produce medically significant results.

Disclosed embodiments of this application address these and other issuesby performing comparison of anatomical images after registration ofimages. The methods and apparatuses of the present invention deform aprior image, for registration to a corresponding current image of anorgan. After the prior and current images have been registered to eachother, the images may be compared to detect differences between theorgans illustrated in the images. In one embodiment, the images arebreast images which are registered whereby one of the images is deformedby applying a registration to specific objects such as suspicious areas,dense tissue areas and fatty tissue areas, with constraints to preservesize and shape of the specific objects. The specific objects may beregions of interest in the breast images, such as cancerous lesions.

SUMMARY OF THE INVENTION

The present invention is directed to methods and apparatuses forprocessing and comparing medical images. According to a first aspect ofthe present invention, an image processing method comprises: accessingdigital image data representing a first medical image and a secondmedical image; registering the second image to the first image using aspecific region preserving registration or specific regions preservingregistration, to obtain a registered second image; and comparing thefirst image and the registered second image.

According to a second aspect of the present invention, an imageprocessing apparatus comprises: an image data input unit for accessingdigital image data representing a first medical image and a secondmedical image; a registration unit for registering the second image tothe first image using a specific area preserving registration, to obtaina registered second image; and a visualization unit for comparing thefirst image and the registered second image.

BRIEF DESCRIPTION OF THE DRAWINGS

Further aspects and advantages of the present invention will becomeapparent upon reading the following detailed description in conjunctionwith the accompanying drawings, in which:

FIG. 1 is a general block diagram of a system including an imageprocessing unit for temporal comparison of mammograms according to anembodiment of the present invention;

FIG. 2 is a block diagram illustrating in more detail aspects of theimage processing unit for temporal comparison of mammograms according toan embodiment of the present invention;

FIG. 3 is a flow diagram illustrating operations performed by an imageprocessing unit for temporal comparison of mammograms according to anembodiment of the present invention illustrated in FIG. 2;

FIG. 4 is a block diagram illustrating an exemplary image processingunit for temporal comparison of mammograms according to an embodiment ofthe present invention illustrated in FIG. 2;

FIG. 5 is a flow diagram illustrating operations performed by an imageprocessing unit for temporal comparison of mammograms according to anembodiment of the present invention illustrated in FIG. 4;

FIG. 6 is a flow diagram illustrating operations performed by an imageoperations unit included in an image processing unit for temporalcomparison of mammograms according to an embodiment of the presentinvention illustrated in FIG. 4;

FIG. 7 is a flow diagram illustrating operations performed by apositional adjustment unit included in an image processing unit fortemporal comparison of mammograms according to an embodiment of thepresent invention illustrated in FIG. 4;

FIG. 8A illustrates an exemplary breast image before translation androtation, and FIG. 8B illustrates the breast image of FIG. 8A afterrigid translation and rotation according to an embodiment of the presentinvention illustrated in FIG. 7;

FIG. 9 is a flow diagram illustrating operations performed by asegmentation unit included in an image processing unit for temporalcomparison of mammograms according to an embodiment of the presentinvention illustrated in FIG. 4;

FIG. 10A illustrates an exemplary breast image, and FIG. 10B illustratesresults of dense segmentation for the breast image of FIG. 10A accordingto an embodiment of the present invention illustrated in FIG. 9;

FIG. 11 is a flow diagram illustrating operations performed by aselective registration unit included in an image processing unit fortemporal comparison of mammograms according to an embodiment of thepresent invention illustrated in FIG. 4;

FIG. 12A illustrates an exemplary breast image with a grid of B-splines,and FIG. 12B illustrates the breast image of FIG. 12A after griddeformation according to an embodiment of the present inventionillustrated in FIG. 11;

FIGS. 12C and 12D illustrate two breast images being registered to oneanother according to an embodiment of the present invention illustratedin FIG. 11; and

FIG. 13A illustrates exemplary corresponding current and prior imageswithout registration, and FIG. 13B illustrates the current image and theprior image of FIG. 13A, after non-rigid mass-and-shape preservingregistration performed according to an embodiment of the presentinvention illustrated in FIG. 4.

DETAILED DESCRIPTION

Aspects of the invention are more specifically set forth in theaccompanying description with reference to the appended figures. FIG. 1is a general block diagram of a system including an image processingunit for temporal comparison of mammograms according to an embodiment ofthe present invention. The system 100 illustrated in FIG. 1 includes thefollowing components: an image input unit 27; an image processing unit37; a display 67; an image output unit 57; a user input unit 77; and aprinting unit 47. Operation of the system 100 in FIG. 1 will becomeapparent from the following discussion.

The image input unit 27 provides digital image data. The digital imagedata may be medical images, such as, for example, mammography images,X-ray images, 3D modalities such as CT, MRI and Tomosynthesis, etc.Image input unit 27 may be one or more of any number of devicesproviding digital image data derived from a radiological film, adiagnostic image, a digital system, etc. Such an input device may be,for example, a scanner for scanning images recorded on a film; a digitalcamera; a digital mammography machine; a recording medium such as aCD-R, a floppy disk, a USB drive, etc.; a database system which storesimages; a network connection; an image processing system that outputsdigital data, such as a computer application that processes images; etc.

The image processing unit 37 receives digital image data from the imageinput unit 27 and performs temporal comparison of mammograms in a mannerdiscussed in detail below. A user, e.g., a radiology specialist at amedical facility, may view the output of image processing unit 37, viadisplay 67 and may input commands to the image processing unit 37 viathe user input unit 77. In the embodiment illustrated in FIG. 1, theuser input unit 77 includes a keyboard 81 and a mouse 82, but otherconventional input devices can also be used.

In addition to performing temporal comparison of mammograms inaccordance with embodiments of the present invention, the imageprocessing unit 37 may perform additional image processing functions inaccordance with commands received from the user input unit 77. Theprinting unit 47 receives the output of the image processing unit 37 orof display 67 and generates a hard copy of the processed image data. Inaddition or as an alternative to generating a hard copy of the output ofthe image processing unit 37 or of display 67, the processed image datamay be returned as an image file, e.g., via a portable recording mediumor via a network (not shown). The output of image processing unit 37 mayalso be sent to image output unit 57 that performs further operations onimage data for various purposes. The image output unit 57 may be amodule that performs further processing of the image data, a databasethat collects and compares images, etc.

FIG. 2 is a block diagram illustrating in more detail aspects of theimage processing unit 37 for temporal comparison of mammograms accordingto an embodiment of the present invention. FIG. 3 is a flow diagramillustrating operations performed by an image processing unit fortemporal comparison of mammograms according to an embodiment of thepresent invention illustrated in FIG. 2.

To obtain diagnostic results from mammography images, temporalcomparison of mammograms may be performed. Manual comparison of breastimages without any prior alignment may not be sensitive enough to detectsubtle differences between images, because of the large eye movementneeded to survey breast images arranged side by side. Furthermore, it isdifficult to arrange side by side more than three temporal cases. Forthis reason, digital processing of breast images is highly desirable.

Temporal subtraction of images in mammography is a difficult taskbecause of shape variation between breasts or between the same breastimaged at different times, unusual or abnormal breast shapes, lightingvariations in medical images taken at different times, patientpositioning differences with respect to the mammography machine,variability of breast borders, unclear areas, non-uniform backgroundregions, tags, labels, or scratches present in mammography images, etc.Hence, mammograms from the same patient, taken at different times canappear different for many reasons, including differences in positioningand compression, changes in the breasts, presence and/or progress ofdisease, etc. If temporal mammograms are compared without any attempt atalignment, comparison may be of little use, because extensivedifferences between mammograms may obscure or destroy medicallysignificant differences.

The present invention implements methods and apparatuses for temporalcomparison of breast images using dense-tissue preserving registrationof images, followed by image comparison. The present invention firstregisters breast images to reduce differences between the breasts in theimages, such differences being caused by positioning and compression ofbreasts. The registered breast images may then be compared, to providean indication of subtle changes and differences between the imagedbreasts caused by cancers, etc. Hence, the registration of images withrespect to each other significantly increases the performance ofsubsequent image comparison operation.

When registration does not preserve dense tissue appearance, the massand shape of an object (for example, of a breast) are distorted toachieve complete alignment between that object and a second object.Hence, typical registration algorithms for images intentionally distortthe appearance of one image, to minimize intensity (appearance)differences between the distorted image and the undistorted image. Whilethis type of alignment and comparison technique may work well fornon-rigid tissue (such as body fat), the technique produces undesirableresults for rigid tissue. Specifically, rigid tissue is distortedcontrary to its anatomical properties. Furthermore, distortion of rigidtissue obscures important anatomical changes in some organs, such asbreasts. For example, distortion of a tumor area in one breast in orderto fit that breast to a second breast may obscure crucial differencesbetween the breasts.

Cancerous lesions in a breast normally show up as bright objects likedense tissues in a mammogram. Cancerous lesions may also deform abreast. Hence, shapes of dense tissues are important in mammogramsespecially for detection of masses and architectural distortions. Densetissue areas in breasts typically appear as high intensities onmammograms. The shapes of such bright objects (e.g., tents signs) inmammograms also contain important information. It is thereforeimportant, during mammogram image registration, to preserve thecharacter and the shape of bright objects and areas as much as possible.

Dark regions in a mammogram are typically fat areas. Distorting theseareas is considered to have a small or negligible impact on detection ofbreast masses.

Based on the above considerations, an apparatus of the present inventionincludes a registration unit 105 and a comparison unit 115, asillustrated in FIG. 2. Registration unit 105 performs registration ofmammograms that preserves dense-tissue (including suspicious areas,dense tissue and pectoral muscles) appearance, and comparison unit 115compares registered mammogram images.

Although the various components of FIG. 2 are illustrated as discreteelements, such an illustration is for ease of explanation and it shouldbe recognized that certain operations of the various components may beperformed by the same physical device, e.g., by one or moremicroprocessors.

The methods and apparatuses of the present invention will be describedbelow in the context of mammography images. The methods and apparatusesof the present invention are also applicable to comparison of left andright breasts.

The methods and apparatuses of the present invention are also applicableto comparison or temporal comparison of medical images of other organsbesides breasts. Such organs may present specific regions used for imageregistration, such as, for example, rigid or dense regions.

When used for breast images, the arrangement of elements for the imageprocessing unit 37 illustrated in FIG. 2 performs non-rigid registrationof breast images that preserves dense-tissue, and image comparison forregistered breast images to observe differences between the registeredimages.

Operation of image processing unit 37 will be next described in thecontext of mammography images, for temporal comparison of images of thesame breast. Image processing unit 37 can also be used for registrationof left and right breasts, or for registration of other organs.

Registration unit 105 receives mammography images from image input unit27 (S183). The mammography images may be two or more images of the samebreast, the images being taken at different times. Registration unit 105registers one breast image to another breast image by preserving massand shape of dense areas (such as, for example, pectoral muscles) in thebreast image, while distorting non-dense areas in the breast image toalign the breast images (S185). Registration unit 105 then sends theregistered breast images to comparison unit 115 which compares theregistered breast images (S187). A user, e.g. a radiologist views theregistered images to determine locations for abnormal or suspectstructures in the breast, etc. User input unit 77 may control comparisonunit 115. Comparison unit 115 may also be incorporated within display67. The display 67 may be controlled to send groups of registered imagesto printing unit 47 or image output unit 57.

Operation of the components included in the image processing unit 37illustrated in FIG. 2 will be next described with reference to FIGS.4-13B.

Registration unit 105 and comparison unit 115 are softwaresystems/applications. Registration unit 105 and comparison unit 115 mayalso be purpose built hardware such as FPGA, ASIC, etc.

FIG. 4 is a block diagram illustrating an exemplary image processingunit 37A for temporal comparison of mammograms according to anembodiment of the present invention illustrated in FIG. 2. FIG. 5 is aflow diagram illustrating operations performed by image processing unit37A for temporal comparison of mammograms according to an embodiment ofthe present invention illustrated in FIG. 4.

The image processing unit 37A registers breast images to enable accuratetemporal comparison of breast images. An important goal of registrationis alignment of fatty area of the breasts, so that differences betweenthe dense areas of the breasts will stand out. The present inventionimplements methods and apparatuses that register a breast image toanother breast image without distorting dense breast areas.

A mammographic pose is a particular position of a breast that is imagedon sensors including X-ray films, digital detectors and/or an imagingplate. Depending on the size of the breast and the position of thepatient with respect to the mammography machine, a prior breast imageand a current breast image may be shifted (translated) or rotated withrespect to each other. These differences between breast images arepositioning differences between past and present pose of the patient inthe imaging system. Differences in compression of the breast during twosuccessive scans will also change the appearance of the breast in theoutput images. This is so even when no anatomical or physiologicalchanges are present in the breast. Differences in the compression andplacement of the breast in a mammography machine arise because thebreast is a soft, deformable tissue. Hence, a variable amount of breasttissue may be captured on the plate of the mammography machine. Forexample, a breast image may show more pectoral muscle, while anotherbreast image may show more skin fold areas. Hence, it is difficult todistinguish between genuine pathological changes and changes of imagingpose in breast images that include positional differences.

The goal of registration in the present invention is reduction ofpositioning and compression differences, as well as other non-anatomicaldifferences and medically insignificant differences between images ofthe same breast, or between images of a left and a right breast, so thatanatomically significant differences between images stand out.

As shown in FIG. 4, image processing unit 37A according to thisembodiment includes: an image operations unit 121; a positionaladjustment unit 131; a segmentation unit 141; a selective registrationunit 151; and a comparison unit 115. Although the various components ofFIG. 4 are illustrated as discrete elements, such an illustration is forease of explanation and it should be recognized that certain operationsof the various components may be performed by the same physical device,e.g., by one or more microprocessors.

Generally, the arrangement of elements for the image processing unit 37Aillustrated in FIG. 4 performs preprocessing and preparation of digitalimage data, positional adjustment of breast images from digital imagedata, segmentation of breasts in the breast images, non-rigid mass andshape preserving registration of breast images, and comparison ofregistered breast images.

Image operations unit 121 receives mammography images from image inputunit 27, and may perform preprocessing and preparation operations on themammography images (S202). Preprocessing and preparation operationsperformed by image operations unit 121 may include resizing, cropping,compression, etc., that change size and/or appearance of the mammographyimages. The mammography images may be, for example, a prior mammogramand a current mammogram of the same breast.

Image operations unit 121 sends preprocessed mammography images topositional adjustment unit 131. Positional adjustment unit 131 correctsfor positional differences between breast images (S206). Segmentationunit 141 performs tissue segmentation in the breast images, to identifyregions of dense breast tissue (S212) including suspicious areas andpectoral muscles. Selective registration unit 151 receives one or moresegmented breast images, and applies a constraint driven deformation toone or both breast images, to register the breast images to each other.The deformation applies rigid deformation to the dense breast regions,and non-rigid deformation to other breast regions (S218). Additionalposition adjustment may be performed during registration, to refineregistration. Comparison unit 115 compares the prior and currentregistered mammograms (S261).

Comparison (visualization) unit 115 may compare images using variousmethods, such as digital processing of registered images. For example,registered images may be subtracted to determine differences betweenthem.

Comparison unit 115 may also toggle images back and forth between theoriginal current mammogram and its corresponding non-rigidly alignedmammogram, on a display. Multiple mammograms may be individuallyregistered to an original current mammogram, in which case comparisonunit 115 may toggle the display back and forth between the originalcurrent mammogram and the multiple corresponding non-rigidly alignedmammograms. Changes, lesions, suspicious masses, etc. are identified bycomparison of original current mammogram to the non-rigidly alignedmammograms, using persistence of vision. With this technique, subtledifferences between prior and current mammogram images stand outvisually, when the same display is toggled back and forth between thetwo breast images. For example, an object found in different locationsin two breast images is easily detected, because the object will appearto be moving during the toggling operation. Changes, lesions, suspiciousmasses, etc. may be then be analyzed, marked on the breast image(s),etc.

Registered breast images, and/or breast image comparisonresults/observed differences may be output to printing unit 47 or imageoutput unit 57 (S271).

Image operations unit 121, positional adjustment unit 131, segmentationunit 141, and selective registration unit 151 are softwaresystems/applications. Image operations unit 121, positional adjustmentunit 131, segmentation unit 141, and selective registration unit 151 mayalso be purpose built hardware such as FPGA, ASIC, etc.

FIG. 6 is a flow diagram illustrating operations performed by an imageoperations unit 121 included in an image processing unit 37A fortemporal comparison of mammograms according to an embodiment of thepresent invention illustrated in FIG. 4.

Image operations unit 121 receives two raw or preprocessed breast imagesfrom image input unit 27 (S302). The breast images may be a prior breastimage and a current breast image. Such mammograms are routinely acquiredfrom patients in hospitals, to diagnose or screen for breast cancer orother abnormalities. Image operations unit 121 may perform preprocessingand preparation operations on the mammography images (S304). Suchpreprocessing and preparation operations may include resizing, cropping,compression, etc., that change size and/or appearance of the mammographyimages.

Mammography images typically show breasts on a background. Thebackground may contain artifacts, tags, markers, etc., indicating theview of the mammogram image acquisition, the patient ID, patientposition, etc. Background interference introduces noise in subsequentprocessing of breast images.

Image operations unit 121 detects the background in one or moremammograms, and then suppresses the background using a breast detectionalgorithm (S310). Alternatively or concurrently, image operations unit121 detects breast borders in one or more mammograms (S311).

Tags, markers, and other background artifacts/obstructions may beremoved by image operations unit 121 in step S310 or S311. To detect andsuppress the background of a mammography image, image operations unit121 may detect the breast borders, or detect the breast and mask thebackground so that background pixels have similar intensity. To detectand suppress the background for a mammography image, image operationsunit 121 may also detect the background without detecting the breast,and then mask the background.

In one exemplary embodiment, the background is zeroed-out so that allbackground pixels have intensity zero.

Image operations unit 121 may perform background detection andsuppression for breast images using methods described in the US PatentApplication titled “Method and Apparatus for Breast Border Detection”,application Ser. No. 11/366,495, by Daniel Russakoff and Akira Hasegawa,filed on Mar. 3, 2006, the entire contents of which are herebyincorporated by reference. Other methods for background detection orbreast border detection may also be used.

With the techniques described in the “Method and Apparatus for BreastBorder Detection”, image pixels that belong to the breast are detected.For this purpose, pixels in a breast image are represented in amulti-dimensional space, such as, for example, a 4-dimensional space,with x-locations of pixels, y-locations of pixels, intensity value ofpixels, and distance of pixels to a reference point. K-means clusteringof pixels is then run in the multi-dimensional pixel representationspace, to obtain clusters for a breast image. In one exemplaryimplementation, K-means clustering divides the group of 4-dimensionalpixel representations into clusters such that a distance metric relativeto the centroids of the clusters is minimized. Cluster merging andconnected components analysis are next performed using relativeintensity measures, brightness pixel values, and cluster size, toidentify a cluster corresponding to the breast in the breast image, aswell as clusters not related to the breast, such as clusters thatinclude image artifacts. Artifacts not related to the breast butconnected to the breast are removed using a chain code, and the breastcontour is joined up using linear approximations. With these techniques,non-uniform background regions, tags, labels, or scratches present in abreast image are removed.

Other clustering methods, or other background suppression methods andbreast border detection methods may also be used by image operationsunit 121.

FIG. 7 is a flow diagram illustrating operations performed by apositional adjustment unit 131 included in an image processing unit 37Afor temporal comparison of mammograms according to an embodiment of thepresent invention illustrated in FIG. 4. Positional adjustment unit 131performs rigid-body registration that corrects for differences in breastpositioning. If the background region in the breast images has beendetected and removed, performance of positional adjustment is improved.Rigid-body registration corrects for X and Y dimensional (2D)translation and rotation between the prior and current breast images(S335).

Rigid-body registration is used to reduce differences in the breastcaused by positioning of the breast. In an exemplary embodiment, therigid registration may proceed in two steps. The first step is anexhaustive 1D search where the breast images are segmented from thebackground and allowed to translate only parallel to the chest-wall sidewith respect to one another, while a correlation coefficient betweenbreast images is monitored. The translation with the highest correlationcoefficient may be chosen as the initialization for the next step, afull rigid body registration where the images are free to translate androtate with respect to each other. A gradient descent approach may beused to obtain convergence of the optimization process. Once theoptimization has converged, gross positioning errors between the twomammograms have been corrected.

2D correction between breast images is performed to improve theregistration algorithm, because subsequent rigid registration is drivenby a search algorithm that measures a correlation coefficient betweenprior and current breast regions. In one embodiment, rigid-bodytransformations performed by positional adjustment unit 131 affect onlytranslation and rotation of images and do not change the size or shapeof the breast, because geometric distance relationships in the breastimages are preserved.

FIG. 8A illustrates an exemplary breast image before translation androtation, and FIG. 8B illustrates the breast image of FIG. 8A afterrigid translation and rotation according to an embodiment of the presentinvention illustrated in FIG. 7. The breast image which is translatedand rotated is called the “moving image” in the present invention. Thebreast image which is not subjected to translation and rotation iscalled “fixed image” herein.

FIG. 9 is a flow diagram illustrating operations performed by asegmentation unit 141 included in an image processing unit 37 fortemporal comparison of mammograms according to an embodiment of thepresent invention illustrated in FIG. 4.

For mammography machines, bright areas represent dense areas in thebreast. There is less X-ray penetration for such areas. Dense areas maycontain significant information about breast changes. The registrationtechnique of the present invention registers two breast images to eachother without changing shape and size of the dense area. The shapeand/or size of the fatty tissue area (dark area in this case) may bechanged.

In order to preserve the dense tissue as a rigid object not deformed byregistration, the location of the dense tissue in the breast isidentified. Segmentation unit 141 performs fatty-dense tissuesegmentation in the breast images, to identify dense tissue and/or fattytissue in the breast (S403). In an exemplary embodiment, thesegmentation is performed only for the moving image (the prior image).The segmentation may also be applied to both images.

Fatty-dense tissue segmentation may be performed using methods describedin U.S. patent application Ser. No. 12/149,566 filed on May 5, 2008 andtitled “Method and Apparatus for Thickness Compensation in MammographicImages”, by Kunlong Gu et al., the entire contents of which are hereinincorporated by reference in their entirety. For breast images that areperipherally enhanced, fatty-dense segmentation in a mammography imagecan be achieved using an expectation-maximization algorithm thatestimates the best Gaussian mixture to fit the distribution ofgray-scale pixel values of the mammography image (S407). For breastimages that have not been enhanced peripherally, other fatty-densealgorithms may be used, to estimate the location of the dense area inthe breast (S409). Tissue segmentation may also be performed using othermethods, such as methods described in the following US PatentApplications: U.S. Patent Application titled “Mass Segmentation UsingMirror Image of Region of Interest”, application Ser. No. 11/642,921, byChao Shi and Daniel Russakoff, filed on 21 Dec. 2006, the entirecontents of which are hereby incorporated by reference; US PatentApplication titled “Method and Apparatus for Detection UsingCluster-Modified Graph Cuts”, application Ser. No. 11/633,571, by HuzefaNeemuchwala, filed on 5 Dec. 2006, the entire contents of which arehereby incorporated by reference; US Patent Application titled “Methodand Apparatus for Detection Using Gradient-Weighted and/orDistance-Weighted Graph Cuts”, application Ser. No. 11/633,534, byHuzefa Neemuchwala, filed on 5 Dec. 2006, the entire contents of whichare hereby incorporated by reference.

Fatty-dense tissue segmentation may also be performed using anintensity-based method for segmenting the dense tissue in the breastregion. In mammography images, the dense tissue pixels concentrate at ahigher intensity than fatty tissue pixels. The overall intensity may bemodeled as a two-mode Gaussian mixture model: p(I)=p₁N(I; μ₁,σ₁)+p₂N(I;μ₂,σ₂), where μ₁<μ₂, where I is the pixel intensity and (p₁, μ₁, σ₁) and(p₂, μ₂, σ₂) are the distribution parameters for the fatty and densetissues, respectively. Expectation maximization may be used forestimating the parameters based on these models, with a techniquesimilar to techniques used in the publication “Nonrigid RegistrationUsing Free-Form Deformations: Application to Breast MR Images”, by D.Rueckert, L. I. Sonoda, C. Hayes, D. L. G. Hill, M. O. Leach, and D. J.Hawkes, IEEE Transactions on Medical Imaging, vol 28, no, 8, pp.712-721, August 1999, the entire contents of which are herebyincorporated by reference. Subsequently, the segmentation threshold(I_(thr)) may be calculated by p₁N(I_(thr); μ₁,σ₁)=p₂N(I_(thr); μ₂,σ₂).Pixels with intensity values higher than I_(thr) are marked as the densetissue. To add an additional layer of protection to the dense tissueregions, the dense tissue results may be dilated to yield a finalsegmentation.

Other segmentation methods may also be used.

FIG. 10A illustrates an exemplary breast image, and FIG. 10B illustratesresults of dense segmentation for the breast image of FIG. 10A accordingto an embodiment of the present invention illustrated in FIG. 9. FIG.10B illustrates a breast mask that identifies dense tissue regions forthe breast in FIG. 10A.

FIG. 11 is a flow diagram illustrating operations performed by aselective registration unit 151 included in an image processing unit 37Afor temporal comparison of mammograms according to an embodiment of thepresent invention illustrated in FIG. 4. Selective registration unit 151performs non-rigid mass-and-shape preserving registration of breastimages. Selective registration unit 151 deforms the prior mammogram byapplying a registration to both the dense-tissue and fatty-tissue areaswith constraints to preserve size and shape of the dense area and matchnon-linear differences between two mammograms.

In an exemplary embodiment, the free-form deformation model described inpublication “Maximum Likelihood from Incomplete Data Via the EMAlgorithm”, by A. Dempter, N. Laird, and D. Rubin, Journal of the RoyalStatistical Society, Series B. vol. 39, no. 1, pp. 1-38, 1977, theentire contents of which are hereby incorporated by reference, may beused to perform a non-rigid registration of the two images. Thisdeformation model consists of a B-spline control point grid whoselocations become the parameters to be optimized. A gradient descentoptimization may be performed, with cross correlation as the similaritymeasure.

As illustrated in FIG. 11, once dense areas in the prior mammogram havebeen detected with a fatty-dense segmentation algorithm, non-rigidmass-and-shape preserving registration can be achieved using non-rigidregistration with a rigidity constraint, for the moving image. In anexemplary embodiment, non-rigid registration is performed using adeformable transformation model such as B-spline, T-spline or finiteelement method. In this exemplary embodiment, a low-resolution grid ofB-spline control points is placed on one of the breast images hereincalled “the moving image”) (S515). The locations of the B-spline gridpoints form the parameters of a search algorithm that identifies abreast deformation function.

In a preferred embodiment, distortion of the dense area in the movingimage is measured using a model related to methods disclosed in“Nonrigid Registration Using a Rigidity Constraint”, by M. Staring, S.Klein and J. P. W. Pluim, presented at SPIE Medical Imaging: ImageProcessing, San Diego, Calif., USA, February 2006, and published inProceedings of SPIE, vol. 6144, pp. 355-364, and/or methods disclosed inthe publication “A Rigidity Penalty Term for Nonrigid Registration”, byM. Staring, S. Klein and J. P. W. Pluim, published in Medical Physics,vol. 34, no. 11, pp. 4098-4108, November 2007, the entire contents ofthese publications being hereby incorporated by reference. The model ofthe present invention measures distortion in the dense area, and thedeformation search is penalized using a weighted distortion measure, topreserve certain differences between mammograms, such as, for example,cancerous lesions. Image similarity is measured as a weighted sum of thecorrelation coefficient between the moving and fixed breast images and apenalty term representing non-linear distortion of the dense area in themoving image. In an exemplary implementation, only B-spline grid pointsthat coincide with dense tissue in the moving image are used to measurethe non-linear deformation.

Hence, the distortion measure implements rigid deformation in denseareas of the moving breast image. Since rigid deformation preservesdistance relationships, measuring the change in distances betweencontrol points helps measure non-rigid distortion.

As mentioned above, one of the breast images received by imageprocessing unit 37 is the moving image, and the other breast image is afixed image. The goal of breast image registration is to identify adeformation u(x) so that the moving image (to which the deformation u(x)is applied) and the fixed image are spatially aligned to each other.

In one embodiment, a distortion measure is obtained using techniquesdescribed in the paper “A Rigidity Penalty Term for NonrigidRegistration”, by M. Staring, S. Klein, and J. P. W. Pluim, Med. Phys.34 (11), November 2007, pp 4098-4108, the entire contents of which ishereby incorporated by reference. With this technique, nonrigidtransformations are locally penalized using a rigidity penalty term(S517) and some parts of the image are restricted to rigid movement,other parts are penalized partially, and other parts are allowed todeform freely (S519).

A deformation function for which the moving image and the fixed imageare spatially aligned to each other is obtained (S522).

In an exemplary embodiment, image similarity may be measured todetermine the deformation function. Image similarity between two imagesI1 and I2 may be written as M(I₁,I₂)=C_(I) ₁ _(,I) ₂ −αρ^(rigid), whereI₁ and I₂ indicate the current and prior images, respectively; C_(I) ₁_(,I) ₂ is a correlation coefficient between pixel intensities of thebreast regions; ρ^(rigid) measures a distortion penalty incurred duringnon-linear deformation of the dense area of the breast; and α is aweighting factor used to help compare the two terms of this measurement.The calculated image similarity between images I₁ and I₂ measures thequality of registration between the images. The search for aregistration deformation function may be stopped when image similaritybetween the fixed image and the iteratively registered moving image isgood enough. For this purpose, cross correlation may be maximized.

The penalty term mentioned above acts on the deformation model. In anexemplary embodiment, the penalty term may consist of three internalpenalties requiring that the deformation be affine, orthonormal andproper. To protect the dense tissue region of the mammogram, the penaltyterm is applied only to the B-spline grid control points inside of thedense region. Mathematically, the algorithm in this exemplary embodimentis described by equation

${v^{*} = {\max_{v}\left( {{{cc}\left( {I_{curr},{T_{v}^{d}\left( I_{prev} \right)}} \right)} - {\alpha{\sum\limits_{v \in {vdense}}{\rho(v)}}}} \right)}},$where I_(curr) is the current mammogram (I₁), I_(prev) is a mammogramtaken at a previous time (I₂), “cc” represent the correlation betweentwo images, T_(v) ^(d) represents the B-spline deformable transformparameterized by v, the set of grid points and ρ represents the rigiditypenalty term applied only to control points inside of the dense tissueregion. The entire penalty term is further scaled by α which attempts tonormalize the contributions from both the cross-correlation term and thepenalty term.

FIG. 12A illustrates an exemplary breast image with a grid of B-splines,and FIG. 12B illustrates the breast image of FIG. 12A after griddeformation according to an embodiment of the present inventionillustrated in FIG. 11. FIGS. 12C and 12D illustrate an exemplary movingimage I₁ and an exemplary fixed image I₂. The grid is placed on themoving image I₁, to perform registration of the moving image I₁ to thefixed image I₂.

FIG. 13A shows exemplary corresponding current and prior images withoutregistration. As shown in FIG. 13A, image displacement prior toregistration can be seen even if the same technologist positions thepatient each year in the mammography machine.

When registration was performed, the prior image was warped into thereference frame of the current image. In FIG. 13B, the same currentimage and the deformed prior image are shown, after non-rigidmass-and-shape preserving registration was performed for the priorimage. As shown in FIG. 13B, the image displacement observed in theimages in FIG. 13A is significantly reduced.

Temporal registration that constrains deformations in breasts, accordingto the present invention, may be achieved using other registrationmodels as well. For example, rigid temporal registration in breasts maybe performed using a model related to techniques disclosed in“Volume-Preserving Non-Rigid Registration of MR Breast Images usingFree-Form Deformation with an Incompressibility Constraint”, by T.Rohlfing, C. R. Maurer Jr., D. A. Bluemke and M. A. Jacobs, published inIEEE Transactions on Medical Imaging, vol. 22 No. 6, pp. 730-741, June2003, the entire contents of which are hereby incorporated by reference.“Volume-Preserving Non-Rigid Registration of MR Breast Images usingFree-Form Deformation with an Incompressibility Constraint” disclosesthe use of an incompressibility constraint as a penalty term, applied toproblems in digital subtraction angiography (DSA) and ComputedTomography (CT) images of the thorax.

The present invention discloses methods and apparatuses for temporalcomparison of registered mammograms or images of other organs. Accordingto the present invention, a non-rigid mammogram registration methodselectively applies rigid (for example, linear) deformations to densetissue, and non-linear deformations to fatty tissue and skin in a breastimage. Dense tissue is aligned by rigid-body transformations, whilefatty tissue is stretched or compressed accordingly. A rigidityconstraint on the dense tissue is used to minimize distortion whilesearching for improved or optimized alignment between the images. Themethods and apparatuses of the present invention can be used to trackanatomic changes in the breast after positioning differences betweenbreast images have been reduced.

In an exemplary embodiment of the present invention, suspect lesions inthe breast are searched by first detecting dense tissue in themammograms. In general, indications of disease such as breast masses andarchitectural distortions show up as dense areas which appear bright onmammograms. Furthermore, the shapes of these bright objects (e.g., tentssigns) also contain important information. When manipulating a mammogramby registration in an exemplary embodiment of the present invention, thecharacter and shape of the bright objects is preserved. On the otherhand, dark regions in mammograms, which are mostly fatty areas, can bedistorted because distorting these areas may have a small impact in thedetection of masses and architectural distortions. A non-rigid mammogramregistration method according to an exemplary embodiment of the presentinvention selectively applies linear deformations, withoutmagnification, to dense tissue, and non-linear deformations to fattytissue and skin, and uses a rigidity constraint on the dense tissue tominimize distortion while searching for the best alignment between theimages.

Methods and apparatuses of the present invention may be used to alignand compare images of the same breast, where the images were taken atdifferent times. For example, images of a breast, taken over a fewyears, can be aligned using methods and apparatuses of the currentinvention, to observe breast shape evolution. Methods and apparatuses ofthe present invention may also be used to align and compare images ofthe left and right breasts of a patient.

Methods and apparatuses of the present invention perform registration ofbreast images to improve visualization of mammograms on digitalworkstations, and help medical specialists effectively compare breastimages. The techniques described in the present invention can alignpairs of mammography images irrespective of pose (CC pairs, ML pairs,etc.); do not need information from ancillary features such as nipple orpectoral muscles; and are not affected by image noise, artifacts,lead-markers, pacemakers or implants.

Methods and apparatuses of the present invention may be used forcomparison/temporal comparison of other medical images besides breastimages. Dense-tissue-preserving registration may be applied to medicalimages of different organs, to compare such medical images and determinechanges in the imaged organs.

Although detailed embodiments and implementations of the presentinvention have been described above, it should be apparent that variousmodifications are possible without departing from the spirit and scopeof the present invention.

1. An image processing method, said method comprising: accessing digitalimage data representing a first medical image and a second medicalimage; segmenting said second medical image to identify a first type oftissue and a second type of tissue; registering said second medicalimage to said first medical image using a specific region preservingregistration or specific regions preserving registration by which massand shape of the first type of tissue is preserved in said secondmedical image, to obtain a registered second medical image; andcomparing said first medical image and said registered second medicalimage.
 2. The image processing method as recited in claim 1, wherein,for one of said first and second medical images, said registering stepperforms rigid-body registration and said segmenting step performs asegmentation of a specific region or regions.
 3. The image processingmethod as recited in claim 1, wherein said registering step includesrigid-body registration to correct for differences in positioning ofanatomical objects in said first and second medical images, using atleast one of translation and rotation.
 4. An image processing method,said method comprising: accessing digital image data representing afirst medical image and a second medical image; registering said secondimage to said first image using a specific region preservingregistration or specific regions preserving registration, to obtain aregistered second image; and comparing said first image and saidregistered second image, wherein said images are breast images and saidregistering step performs rigid-body registration, fatty-dense tissuesegmentation and non-rigid registration preserving dense tissue, forsaid breast images.
 5. The image processing method as recited in claim1, wherein said registering step is driven by as search algorithm thatmeasures a correlation coefficient between said first medical image andsaid registered second medical image.
 6. The image processing method asrecited in claim 1, wherein said first and second medical images arefirst and second breast images, and said registering step deforms saidsecond breast image by applying a registration to specific regions, withconstraints to preserve size and shape of the specific regions.
 7. Theimage processing method as recited in claim 6, wherein said specificregions are cancerous lesions.
 8. The image processing method as recitedin claim 1, wherein said segmenting step includes: identifying specificregions which are dense tissue areas in said second medical image usinga segmentation algorithm, and said registering step includes:non-rigidly registering said second medical image to said first medicalimage using a rigidity constraint to penalize deformation of said densetissue areas.
 9. The image processing method as recited in claim 1,wherein said registering step includes: measuring an image similarityusing a correlation coefficient between said first medical image andsaid registered second medical image, and a penalty term representing adistortion of a specific region, and obtaining a final registered secondmedical image to increase said image similarity.
 10. The imageprocessing method as recited in claim 8, wherein said registering stepfurther includes: applying a deformable transformation model.
 11. Theimage processing method as recited in claim 8, wherein saidtransformation includes conditions to penalize deviation from rigidity.12. The image processing method as recited in claim 1, wherein saidfirst and second medical images are first and second breast images, andsaid segmenting step includes identifying dense tissue by a fatty-densetissue segmentation for said second breast image and/or said firstbreast image.
 13. The image processing method as recited in claim 1,wherein said comparing step is used to compare said first medical imageto said registered second medical image using persistence of vision bytoggling images on a display.
 14. An image processing apparatus, saidapparatus comprising: an image data input unit for accessing digitalimage data representing, a first medical image and a second medicalimage; a registration unit for segmenting said second medical image toidentify a first type of tissue and a second type of tissue, andregistering said second medical image to said first medical image usinga specific region preserving registration by which mass and shape of thefirst type of tissue is preserved in said second medical image, toobtain a registered second medical image; and a visualization unit forcomparing said first medical image and said registered second medicalimage.
 15. The apparatus according to claim 14, wherein saidvisualization unit compares said first medical image to said registeredsecond medical image and detects differences between images.
 16. Theapparatus according to claim 14, wherein said registration unit performsrigid-body registration to correct for differences in said first andsecond medical images, using at least one of translation and rotation.17. The apparatus according to claim 14, further comprising an imageoperations unit for detecting and ignoring specific regions in saidsecond medical image.
 18. The apparatus according to claim 14, whereinsaid registration unit is driven by a search algorithm that measures acorrelation coefficient between said first medical image and saidregistered second medical image.
 19. The apparatus according to claim14, wherein said first and second images are first and second breastimages, and said registration unit deforms said second breast image byapplying a registration to specific regions in said second breast image,with constraints to preserve size and shape of the areas.
 20. An imageprocessing apparatus, said apparatus comprising: an image data inputunit for accessing digital image data representing a first medical imageand a second medical image; a registration unit for registering saidsecond image to said first image using a specific region preservingregistration, to obtain a registered second image; and a visualizationunit for comparing said first image and said registered second image,wherein said registration unit identifies dense tissue areas in saidsecond image using a segmentation algorithm, and non-rigidly registerssaid second image to said first image using a rigidity constraint topenalize deformation of said dense tissue areas.
 21. The apparatusaccording to claim 19, wherein said registration unit measures an imagesimilarity using a correlation coefficient between said first medicalimage and said registered second medical image, and a penalty termrepresenting a distortion of said dense tissue, and obtains a finalregistered second medical image by increasing said image similarity. 22.The apparatus according to claim 21, wherein said registration unitapplies a deformable transformation model.
 23. The apparatus accordingto claim 15, wherein said first and second medical images are breastimages of the same breast taken at different times, and saidvisualization unit is used to visually compare said breast images todetermine changes in the breast images.
 24. The apparatus according toclaim 19, wherein said registration unit identifies dense tissue by afatty-dense tissue segmentation for said second breast image and/or saidfirst breast image.
 25. The apparatus according to claim 14, whereinsaid visualization unit compares said first medical image to saidregistered second medical image using, persistence of vision throughimage toggling.
 26. The image processing method as recited in claim 6,wherein said specific region include one or more of suspicious areas,pectoral muscle and dense tissue areas.
 27. The image processing methodas recited in claim 9, wherein said specific region is dense tissue. 28.The image processing method as recited in claim 10, wherein saiddeformable transformation model is one of a B-spline, T-spline or finiteelement method.
 29. The apparatus according to claim 19, wherein saidspecific regions are suspicious areas, pectoral muscle or dense tissueareas.
 30. The apparatus according to claim 22, wherein said deformabletransformation model is one of a B-spline, T-spline or finite elementmethod.